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On  the  Correspondence  Between  'Primary'  and  'Secondary' 
Measures  of  Business  Economic  Performance: 
An  Attempt  at  Methodological  Triangulation 


Abstract 

This  study  is  an  attempt  at  assessing  method  convergence  between 
two  different  operationalizations  of  Business  Economic  Performance — 
viz.,  managers'  assessment  of  their  organization's  relative  perfor- 
mance and  secondary  analysis  based  on  published  sources.   An 
evaluation  of  the  convergence  of  three  indicators  of  performance — 
sales  growth,  profit  growth,  and  ROI — provided  strong  results,  indi- 
cating that  managers  are  generally  not  biased  in  their  assessments. 
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Introduction 

Concern  with  enhancing  organizational  performance  is  at  the  heart 
or  most  organizational  research  (Campbell,  1977;  Chakravarthy ,  1984; 
Ford  oc    Schellenberg,  1982;  Hofer,  1983;  Kanter  6<  3ri  nkerhof  f ,  1981; 
Kirchoff,  1977;  Seashore  &  Yuchtman,  1967;  Steers,  1975,  1977). 
Perhaps  more  than  any  other  branch  of  the  organizational  sciences,  the 
field  of  strategic  management  is  centrally  focused  on  issues  of  orga- 
nizational performance  (Schendel  &  Hofer,  1979).   A  normative  theory 
of  strategic  management  awaits  not  only  the  clarification  of  the  term 
"strategic  performance,"  but  also  the  development  of  reliable  and 
valid  approaches  to  the  measurement  of  the  performance  construct. 

Two  major  issues  need  to  be  addressed  in  dealing  with  the  opera- 
tionalization  of  performance  construct.   One  pertains  to  the  choice  of 
an  appropriate  set  of  operational  indicators  reflecting  the  construct's 
domain,  while  the  other  is  concerned  with  the  method  of  data  collection 
The  first  issue  is  plagued  with  controversies  and  debates  (see  espe- 
cially Campbell,  1977;  Connolly,  Conlon,  &  Deutsch,  1980;  Goodman  Ct 
Pennings,  1977;  Hanna,  Freeman,  &  Meyer,  1976;  Steers,  1975;  1977), 
and  we  will  have  little  to  say  on  this  here.   In  relation  to  tne  second 
issue  researchers  typically  are  faced  with  the  choice  of  obtaining 
performance  data  either  from  'secondary*  sources — i.e.,  data  collected 
from  sources  external  to  the  organization,  or  from  'primary'  sources — 
i.e.,  data  collected  from  the  organizations  themselves.   While  opera- 
tionalizations  based  on  secondary  data  permit  replicabili ty ,  primary 


data  could  introduce  respondent  bias  and  nay  not  serve  the  interests  of 
replicabi li ty .   On  the  other  hand,  secondary  data  on  performance  may 
not  be  available  at  the  desired  level  of  detail  for  some  applications 
(e.g.,  for  an  SBU-level  focus). 

.lost  researchers  choose  one  of  these  approaches  but  seldom  provide 
evidence  of  convergence  witn  the  other  operationalization.   Given  that 
both  approaches,  wnen  considered  individually,  may  have  questionable 
measurement  properties,  it  is  necessary  to  address  the  issue  of  "method 
convergence"  (Campbell  &  Fiske,  1959)  in  operationalizing  organiza- 
tional performance  to  ensure  that  the  variance  reflected  is  that  of 
trait  and  not  of  method.   Such  attempts  reflect  the  philosophy  under- 
lying methodological  triangulation  (Denzin,  1978;  Jick.,  1979) — the  use 
of  complementary  methods  to  enhance  researcher's  beiief  in  results. 

Thus,  this  study  seeks  to  assess  the  extent  of  convergence  in 
operationalizations  of  the  construct  of  Business  Economic  Performance 
(BEP)  by  collecting  data  from  two  different  methods — viz.,  primary  data 
from  organizations  themselves,  and  secondary  data  from  published  sources, 
external  to  the  organization.   Such  an  approach  reflects  Campbell  and 
Fiske's  (1959)  call  for  using  'maximally  different'  methods  to  assess 
convergent  validity  of  operationalizations.   As  noted  by  Bouchard,  con- 
vergence between  two  methods  "ennances  our  belief  that  the  results  are 
valid  and  not  a  methodological  artifact"  (1976;  p.  268).   When  maximally- 
differing  methods  are  used,  the  approach  is  termed  ' between-raethods' 
triangulation  (Denzin,  1978;  Jick,  1979),  which  rests  on  the  assumption 
that  the  two  methods  do  not  share  the  same  weakness  or  potential  bias 
(Rohner,  1977). 
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Thus ,  this  research  attempts  a  methodological  triangulation  of  the 
correspondence  between  two  different  operationalizations  of  BEP — 
managers'  assessment  of  their  organization's  relative  competitive  per- 
formance and  secondary  analysis  of  relative  performance  based  on 
published  sources. 

Research  Method 


Indicators  of  BEP:   Three  indicators — viz.,  sales  growth,  net 
income  growth,  and  return  on  investment  (ROI) — were  chosen  to  reflect 
BEP.   These  three  indicators  correspond  to  the  key  dimensions  of  per- 
formance distilled  by  Woo  and  Willard  (1983)  based  on  their  analysis 
of  PIMS  data — viz.,  (i)  profitability;  (ii)  relative  market  position; 
(iii)  change  in  profitability,  and  (iv)  growth  in  sales  and  market 
share.   Hofer  (1983)  also  found  these  indicators  to  be  among  the  most 
commonly  used  measures  of  BEP.   Hence,  an  examination  of  the  method 
convergence  of  these  indicators  should  be  of  interest  to  strategy 
researchers  operationalizing  business  performance. 

Primary  measures:   For  each  of  the  three  indicators,  managers  were 
requested  to  indicate  their  positions,  not  of  their  absolute  perfor- 
mance but  their  performance  relative  to  their  major  competitors.   This 
reflects  the  "relative"  nature  of  the  performance  concept  stressed  by 
many,  including  the  PIMS-based  strategy  studies.   A  five-point  inter- 
val scale  ranging  from  -2  (much  worse  than  competition)  to  +2  (much 
better  than  competition)  with  the  neutral  point  0  indicating  a  level 
of  performance  equal  to  that  of  competition  was  employed.   Data  were 
collected  from  senior-level  managers  (either  presidents/vice  presi- 
dents of  functional  areas  or  vice  presidents  of  corporate  planning)  as 


a  part  of  a  larger  project  during  Feoruary-May  1984.   Alchough  the 
larger  project  had  a  response  rate  of  over  33£  (207  out  of  bOO),  only 
8b  cases  are  used  in  this  study.   Since  anonymity  was  to  be  ensured, 
the  respondent's  name,  and  corporate  affiliation  was  voluntary.   8b 
respondents  indicated  their  organizational  affiliations  which  was 
necessary  to  collect  secondary  data  on  them.   Table  1  lists  some  key 
characteristics  of  the  sample  employed  in  this  study. 


INSERT  TABLE  1  AiiOUT  HERE 


Secondary  measures:   For  each  of  the  three  indicators,  secondary 
measures  were  assembled  from  Business  Week  magazine's  "Inflation 
Scorecard"  for  the  year  1983,  as  reported  in  the  March  21,  1984  issue. 
Business  Week  compiles  these  data  from  Standard  &  Poor's  COMPUSTAT 
tapes,  and  was  a  convenient  and  easily  accessible  source  of  data. 
Relative  performance  was  operationalized  as  "firm  performance  relative 
to  industry" — where  industry  referred  to  the  principal  SIC  industry 
classification  in  which  the  firm  was  normally  placed.   It  was  measured 
as  the  difference  between  the  value  of  the  indicator  for  the  firm  and 
the  industry.   For  example,  relative  sales  growth  was  the  sales  growth 
of  the  focal  firm  minus  the  sales  growth  of  its  primary  industry. 

Results 


Table  2  presents  the  descriptive  statistics  as  well  as  the  analy- 
sis in  the  form  of  Campbell  and  Fiske's  MultiTrait,  MultiMethod  (MTMM) 
matrix  which  is  one  of  the  analytical  schemes  of  methodological  triangu- 
lation.   Entries  in  the  MTMM  matrix  are  Pearson's  Zero-order  correlations. 
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INSERT  TABLE  2  ABOUT  riERE 

The  first  of  Che  four  criteria  of  an  MTMiM  matrix  (Campbell  &  Fisfce, 
1959)  refers  to  convergent  validity  and  requires  that  all  the  diagonal 
coefficients  in  the  lower  left  quadrant  of  the  matrix  (termed,  "validity 
coefficients")  be  "sufficiently  large"  and  statistically  significant 
(Campbell  6  Fiske,  1959).   Table  2  indicates  that  all  the  three  valid- 
ity coefficients  are  greater  than  0.4  and  statistically  significant 
at  a  p-level  better  than  0.01. 

The  other  three  criteria  relate  to  discriminant  validity,  viz., 
whether  the  three  traits  are  different  from  one  another  or  not.   While 
these  criteria  are  not  directly  relevant  for  the  attempt  at  checking 
for  correspondence,  they  imply  that  measures  of  different  concepts 
should  share  little  common  variance,  since  a  high  level  of  covariation 
casts  doubt  on  the  uniqueness  of  measures  and/or  the  concepts.   The 
second  criterion  requires  tnat  each  validity  coefficient  should  be 
larger  than  the  "different  trait-different  method"  correlations  (which 
are  in  the  same  row  or  column  as  the  validity  coefficients  in  the 
dashed  triangles  adjacent  to  the  validity  coefficient).   As  shown  in 
Table  2,  this  condition  is  satisfied  in  all  three  cases. 

The  third  criterion  requires  that  each  validity  coefficient  should 
be  larger  than  the  "different  trait-same  method"  correlations  (which 
involve  tne  same  variable  as  that  of  the  validity  coefficient  in  the 
lower  right  and  upper  left  quadrants).   This  condition  is  satisfied  in 
two  of  the  four  cases  for  the  sales  growth  measure,  in  one  out  of  the 
four  cases  for  tne  profit  growth  measure,  and  in  three  out  of  the  four 
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cases  for  the  RUI  measure.   The  general  support  for  this  criterion 
appears  to  be  "moderate." 

The  fourth  and  final  criterion  requires  that  the  pattern  of  corre- 
lations present  in  each  of  the  four  triangles  (both  solid  and  dashed) 
in  the  matrix  should  be  similar.   A  test  of  this  similarity  can  be 
accomplished  by  ranking  the  correlations  in  each  triangle  and  deriving 
a  measure  of  the  rank,  correlation  across  the  triangles.   J-'riedman  two- 
way  test  was  conducted  for  this  purpose.   Its  associated  chi-squared 
statistic  was  6.50  (df=2),  statistically  significant  at  a  _p  value  of 
0.039.   Thus,  we  conclude  that  the  relative  rankings  of  the  correla- 
tions is  preserved  within  the  four  triangles,  thereoy  satisfying  the 
fourth  criterion. 

Discussion 

The  results  (especially,  the  first  criterion  of  the  MTMiM  matrix) 
indicate  that  there  exists  a  strong  degree  of  "method  convergence"  when 
performance  data  was  obtained  from  two  'maximally  different'  methods. 
It  appears  that  respondents  tend  to  be  less  biased  in  their  assessments 
of  their  organizational  performance  than  researchers  have  tended  to 
give  them  credit  for.   The  main  implication  of  our  finding  is  that  per- 
ceptual data  from  senior  managers,  which  tend  to  correlate  well  with 
secondary  data,  can  be  employed  as  acceptable  operationalizations  of 
BEP. 

A  previous  study  by  Dess  and  Robinson  (1984),  using  self-reported 
'objective'  data  and  suDjective  assessments  of  two  performance 
ndicators — return  on  assets  and  sales  growtn,  reported  a  close  corre- 

dence  between  the  two  operationalizations.   Their  two  approaches 
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are  conceptually  similar  in  the  sense  of  employing  data  collected  from 
only  primary  source,  and  represent  'within-method '  type  of  triangulation 
(Denzin,  1978).   The  limitations  of  this  type  of  triangulation  are  noted 
by  Denzin,  "Observers  delude  themselves  into  believing  that .. .different 
variations  of  the  same  method  generate .. .distinct  varieties  of  triangu- 
lated data.   But  the  flaws  that  arise  using  one  method  remain...." 
(1978,  pp.  3U1-302).   in  contrast,  the  present  study,  wnich  reflects  a 
' between-method '  type  of  triangulation  rests  on  the  assumption  that  the 
weaknesses  in  each  single  method  will  be  compensated  oy  the  counter- 
balancing strengths  of  another.   Thus,  this  study  can  De  seen  as  a  study 
which  moves  the  operationalizations  of  BEP  towards  the  'between-methods ' 
approach  to  triangulation,  which  "allows  researchers  to  be  more  confi- 
dent of  their  results"  (Jick,  1979;  p.  608). 

Although  the  study  established  correspondence  across  two  maximally 
different  methods,  a  potential  limitation  of  this  study  should  be 
recognized.   Data  for  this  study  were  collected  from  a  single  respondent 
in  each  responding  unit.   Hence,  the  possibility  of  functional  or 
response  bias  cannot  be  entirely  ruled  out.   It  would  have  been  desir- 
able to  collect  data  from  multiple  managers  within  a  unit  so  that  inter- 
manager  consistency  could  have  been  assessed.   However,  the  size  of  the 
target  population  and  resource  limitations  prompted  us  to  trade  off  in 
favor  of  larger  sample  size  rather  than  multiple  responses  per  unit. 
Further,  based  on  results  obtained  in  earlier  studies  which  have  employed 
tne  "multiple  respondent  design"  (e.g.,  Dess  &  Robinson,  1984;  Snow  & 
Hreoiniak,  1980)  it  can  be  argued  tnat  tnere  is  generally  less  variability 
within  raters  of  a  particular  firm  than  raters  across  firms. 
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Moreover,  respondents  in  our  sCudy  were  senior-level  managers 
(e.g.,  vice  president-strategic  planning,  president  or  functional  vice 
president)  who  can  be  argued  to  be  key  members  of  the  dominant  coali- 
tion of  tne  firm,  and  tnus  can  be  considered  as  "'representatives"  of 
the  organization.   Thus,  while  Uess  and  Robinson's  study  addressed  the 
measurement  theme  of  inter-judge  reliability  in  performance  assessment, 
this  study  focused  on  a  different  measurement  issue,  viz.,  convergence 
across  "maximally  different  methods" — which  is  a  key  requirement  for 
construct  validity  of  measures  (Campbell  6  Fiske,  19 59 )  .   Nevertheless, 
the  issue  of  using  multiple-respondents  to  measure  organizational-level 
constructs  such  as  BEP  needs  to  be  addressed  by  strategy  researchers. 

The  use  of  MTMM  framework,  enabled  us  to  address  a  related  issue  of 
"uniqueness"  of  the  tnree  traits  considered  for  operationalizing  BEP. 
The  support  received  for  the  three  criteria  of  discriminant  validity 
imply  that  the  tnree  indicators  considered  here  tap  different  "traits" 
of  SEP.   This  is  in  agreement  with  the  findings  of  Woo  and  Millard 
(1983) — who  employed  a  different  data-analytic  framework  (factor  analy- 
sis) and  a  different  data  base  (the  PIMS  program)  in  arguing  for  a 
multi-dimensional  operationalization  of  performance.   However,  the 
results  of  this  study  should  not  be  taken  to  indicate  that  these  are 
the  only  dimensions.   Nor  is  it  implied  by  us  that  these  are  tne  key 
dimensions  of  organizational  performance.   Based  on  the  results  reported 
here  and  previous  theoretical  arguments  (e.g.,  Campbell,  1977;  Steers, 

75;  1977)  and  empirical  results  (e.g.,  Woo  &  Willard,  1983),  we  argue 
that  the  use  of  any  single  indicator  (dimension)  to  capture  the  rela- 
ely  complex  construct  of  performance  should  be  viewed  with  disfavor. 
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hxtensions 

Measurement  of  organizational  performance  in  general,  and  BEP  in 
particular  is  central  to  research  in  strategic  uanageraent .   Towards 
this  end,  Woo  and  Willard's  (19tf3)  study,  Dess  and  Kooinson's  (1984) 
study  and  the  results  reported  here  are  to  be  viewed  as  starting  points 
for  further  refinement  and  extensions.   Future  research  directions  on 
performance  measurement  can  be  broadly  grouped  under  three  streams. 
One  is  to  employ  multiple  managers  cnosen  to  represent  different  func- 
tional areas,  hierarchical  level,  or  length  of  tenure  with  the  company, 
since  these  variables  may  have  an  impact  on  the  aDility  of  respondents 
to  make  complex  judgments  on  assessing  organizational  performance. 
Given  managers'  differing  f rames-of-ref erence ,  such  an  analysis  could 
provide  interesting  and  useful  pointers  for  the  choice  of  respondents 
in  the  design  of  field  studies.   Two ,  since  the  domain  of  organiza- 
tional performance  extends  beyond  8EP,  similar  methodological  triangula- 
tion  attempts  to  assess  convergence  should  be  undertaken  for  broader 
conceptualizations  of  organizational  performance  which  include  both 
financial  and  operational  indicators. 

The  third  stream  of  extension  relates  to  the  issue  of  superiority 
of  one  operationalization  over  another.   This  is  important  since 
researchers  examining  convergence  between  methods  to  assess  the 
quality  of  their  operationalizations  may  elect  to  use  one  or  the 
other,  but  not  necessarily  both.   In  this  context,  Dess  and  Robinson 
in  concluding  their  study  expressed  their  preference  for  'objective' 
data  by  noting  that  subjective  performance  data  are  good  substitutes 
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for  objective  data  whenever  "(.J.)  accurate  objective  measures  are  un- 
available, and  (2)  trie  alternative  is  to  remove  tne  consideration  of 
performance  from  the  research  design"  (1984,  p.  271).   This  was  not 
based  on  any  specific  analysis  of  the  super iori  ty  of  one  method  over 
another,  but  merely  reflects  their  note  of  caution.   It  should  prove 
useful  to  systematically  assess  the  relative  superiority  of  one  metnod 
over  another,  by  analytical  approaches  such  as  the  analysis  of  co- 
variance  structures  (Joreskog  &  Sorbum,  1979),  which  provides  a  basis 
to  decompose  the  variance  in  measurement  into  key  components  such  as 
trait,  method,  and  random  error. 

Strategy  researchers  in  particular  need  to  take  cognizance  of 
these  conceptual  and  measurement  issues  in  view  of  tne  embryonic  nature 
of  their  field.   Although  a  paradigm  of  strategic  management  is  at  hand 
(Schendel  &  Hofer,  1979),  a  normative  theory  of  strategic  management  can- 
not be  developed  unless  the  crucial  issues  of  conceptualizing  and  measur- 
ing organizational  performance  are  more  fully  researched  and  understood. 

Sumraarv 


Data  on  three  commonly  employed  indicators  of  performance — sales 
growth,  profit  growth,  and  ROI — were  collected  by  two  different 
methods — (i)  perceptual  assessments  by  senior  executives  and  (ii) 
secondary  data  sources.   An  evaluation  of  their  convergence  provided 
positive  results,  indicating  that  managers  are  generally  not  biased  in 
their  assessments  of  organizational  performance.   In  addition,  it  was 
observed  that  those  indicators  tap  different  traits  of  performance, 
thus  raising  some  important  measurement  issues  in  relation  to  the 
Lmensionality  of  organizational  performance. 
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Table  1 

Key  Characteristics  of  the  Sasple  (n=8b) 

1 .  Sales    level 

$    5U   -    100    Million 

$101    -   250   Million 

$251  -  500  Million 

$501  Million  -  $  1  Billion 

over  $  1  Billion 

2.  Industry   Category 

Consumer   Goods  21.2 

Capital    Goods  31.8 

Raw   or    serai-finished   materials  22.4 

Components    for    finished   goods  12.9 

Service  11.8 

3.  Respondent's    Responsibility 

Staff  responsibility  (e.g.,  V.P.  - 

Strategic  Planning)  79.1 

Operating  responsibility  (e.g., 

President  or  Functional  V.P.s)  20.9 


a 
All  figures  are  percentages.   Non-responses  are  excluded  in 

the  percentage  calculations. 
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Table  2 

Primary  Versus  Secondary  Measures  of 
Business  Economic  Performance:   An  MTMM  Analysis 


PRIMARY 


SECONDARY 


DESCRIPTIVE 
STATISTICS 


Primary 

SGa           PC           RO [ 

SG                PG                RO  I 

MEAN 

SD 

SC 
PG 
ROI 

L.OO 

0.47\    1.00 

0.36        0.74\l.0O 

3.32 

3.47 
3.24 

0.91 
1.05 
1.06 

Secondary 

— 

SG         kp«44s  v0,  i4      0.151 

1    N              N                     ' 

1        \              v                 1 

PG        '  0.32  s    0.42M).33! 
1               \             x      , 

'                    \         N 
ROI      |0.10        0.3b  n0. 51 

U.OO 

0.69    \  1.00 
0.02          0.28\   1.00 

1.13 
(-)2.64 
(-)0.54 

12.54 

21.71 

9.96 

_ 

,— ■ 

__^_____^______ __^ 

Convergent  Validity:  Criterion  1 


All  validity  coefficients  (SG:   0.44;  PG;  0.42;  ROI:   0.51)  are  statistically 
significant  at  p<0.01. 

Discriminant  Validitv:   Criteria  2  and  3 


Validity 
Coefficient 

0.44 

0.42 

0.51 

Criterion   4: 


Criterion    2 
Z    satisfied 

100 

100 

100 


Criterion  3 
%    satisfied 

50 

25 

75 


Chi-squared  statistic  for  Friedman's  non-parametric  test  for  the  rankings 
of  the  correlations  within  the  four  triangles:   6.50  (df=2) ,  statistically 
significant  at  p=0.039. 

Sales  growth;  PG:   Profit  (net  income)  growth;  ROI:   Return  on 
itment.   Entries  in  the  matrix  are  Pearson's  zero-order  correlations. 
mary  data  are  based  on  five-point  Likert-type  scale,  secondary  data 
are  actual  values. 


HECKMAN      |±| 
BINDERY  INC.        |g| 

JUN95 

tmd -To  .PIM5(?  N.MANCHESTER 
INDIANA  46962     ' 


